Challenges in data compression for current and future imagers and hyperspectral sounders

For current high-volume datasets such as IASI, near-real-time dissemination of the fullspectrum presents a challenge, but is nevertheless achievable within current bandwidth constraints, e.g. on EUMETCast. However, this may not be the case for data from future instruments. In this paper we will look at the principles for efficient data compression. It is shown that appropriate normalisation of the spectrum (usually by the instrument noise) is a key to efficient compression – with implications for the design of level 1 data formats. EUMETSAT’s compact VIIRS format is a good example of a format that has been designed with compression and dissemination in mind. Format design should be considered as a key part of the mission planning process – i.e. efficient dissemination formats should be available well before launch. Principal Component (PC) compression is well known in the IASI community, and has the potential to separate signal and noise – though the technique has not so far been widely exploited in operations. Furthermore, PC compression is EUMETSAT’s baseline for near-real-time dissemination of MTG-IRS data. Simulations of MTG-IRS are shown, which illustrate some possible tradeoffs between data volume and reconstruction accuracy.